Exploiting Space-Time Statistics of Videos for “Hallucination“

In this work, we address the task of enhancing the spatial resolution of video sequences, known as super-resolution. Specifically, we consider the problem of superresolving a human face video by a very high (×16) zoom factor. Inspired by recent literature on hallucination and example-based learning, we formulate this task using a graphical model that encodes 1) spatio-temporal consistencies, and 2) image formation & degradation processes. A video database of facial expressions is used to learn a domain-specific prior for high-resolution videos. The problem is now one of probabilistic inference, in which we aim to find the high resolution video that best satisfies the constraints expressed through the graphical model. Traditional approaches to this problem using video data first estimate the relative motion between frames and then compensate for it, resulting effectively in multiple measurements of the scene. Our use of time is rather direct: We define data structures that span multiple consecutive frames, enriching our feature vectors with a temporal signature. We then exploit these signatures to find consistent solutions over time. We present preliminary results in which a 8 × 6 pixel-wide face video, subject to translational jitter and additive noise, gets magnified to a 128 × 96 pixel video. Preliminary results show that, by exploiting both space and time, drastic improvements can be achieved in reducing both video flicker artifacts and mean-squared-error. Proposed extensions include data-driven modifications of the graphical model and improvements on feature definitions as well as compatibility function designs. Finally, the task of human silhouette enhancement in videos is proposed to test our framework as a general, model-based regularization tool.